Infinite Horizon Problems

A typical discrete-time sequential decision problem involves a system whose state is assumed to evolve either deterministically or probabilistically over time-periods that are often called stages. This evolution is affected by the decisions a planner makes at the beginning of each stage after observing the system state. The decision maker’s goal then is to optimize some measure of system performance over a certain time-horizon. For example, in a typical production and inventory management problem, the system state is given by the inventory on hand at the beginning of a stage, and the decision corresponds to the production level in that stage. The inventory beginning the next stage equals the old inventory plus the production quantity minus the stochastic demand filled. Unsatisfied demand may be lost. The planner’s goal may be to maximize the expected total discounted profit, where revenue is generated by selling the product, and costs are incurred for production, inventory holding, and shortage. The dynamic systems in such optimization problems often do not have a predetermined time of extinction. Thus, using a finite planning horizon typically introduces endof-study effects on early decisions. Indeed, a finite horizon formulation of a production planning problem essentially amounts to assuming that the demand in all subsequent time-periods after this horizon is zero. Then the decision maker is likely to plan initial production such that the inventory ending this finite horizon is zero, forcing him/her to produce additional units later when the actual demand beyond the initial study horizon is